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Objective  and  Aims 

This  final  report  briefly  reviews  the  objective  and  specific  aims  of  this  MURI  project, 
then  summarizes  our  progress  toward  these  aims.  The  specific  objective  of  this 
project,  as  stated  in  the  application,  was  to  develop  a  "prototype  of  a  system  for 
communication  and  monitoring  of  orientation  that  uses  brain  signals  to  provide,  in 
real  time,  an  accurate  assessment  of  the  user’s  intentional  focus,  eye  movements, 
and  imagined  speech."  This  project  thereby  sought  to  facilitate  the  DoD's  mission  by 
providing  assessments  that  could  eventually  lead  to  improvements  or 
augmentations  a  soldier’s  performance. 

The  three  specific  aims  were: 


1.  To  delineate  the  brain  signal  features  associated  with  the  direction  of 
attention,  intention,  and  imagined  speech. 

2.  To  determine  whether  these  features  can  be  detected  non-invasively. 

3.  To  validate  the  use  of  brain  signals  for  communication  and  orientation. 


Specific  Aim  One 

We  achieved  substantial  progress  in  delineating  brain  signal  features  associated 
with  all  three  intended  cognitive  functions:  direction  of  attention,  intention,  and 
imagined  speech. 


Attention 

Our  efforts  to  explore  direction  of  intention  included  work  with  auditory  and  visual 
paradigms.  In  2009,  we  began  using  ECoG  to  provide  greater  spatial  resolution  and 
more  detailed  information  than  noninvasive  methods  can  provide.  Five  subjects 
performed  a  visual  task  with  unpredictable  changes  in  peripheral  stimuli.  We 
analyzed  results  with  two  goals:  assessing  key  features  reflecting  neural  correlates 
of  attention;  and  assessing  the  efficacy  of  using  ECoG  to  infer  different  aspects  of 
attention  (active  attention  vs.  rest  and  different  directions  of  attention).  We  first 
identified  key  parietal  and  prefrontal  activities  reflecting  attentional  processes,  and 
showed  a  high  classification  accuracy  with  three  of  the  five  subjects.  In  2010,  we 
extended  this  work  with  additional  research  on  visual  spatial  attention  involving 
covert  attention  to  cued  stimuli.  We  again  capitalized  on  the  advantages  of  ECoG- 
based  methods  to  provide  new  information  about  neural  activity  relating  to 


attention,  in  visual,  motor  and  prefrontal  areas.  We  extended  this  work  in  2011  with 
additional  results  showing  classification  of  attentional  activity  well  above  chance.  In 
2012,  we  conducted  additional  studies  exploring  auditory  and  visual  attention.  For 
example,  in  one  study,  subjects  listened  to  two  speeches  simultaneously  -  the 
inauguration  speeches  of  Presidents  Kennedy  and  Obama.  We  showed  that  we  can 
distinguish  which  of  these  two  binaurally  presented  conversations  that  a  subject  is 
attending.  In  2013,  we  further  explored  data  collected  from  our  attentional  research 
and  found  new  ways  to  improve  classification  activity  and  understand  some  neural 
dynamics  underlying  attention. 


Intention 

Intention  is  linked  to  attention  and  orientation,  and  some  of  our  early  work  in  this 
project  focused  on  intentional  tasks  relating  to  attention.  In  2009,  we  collected  data 
in  which  subjects  performed  a  center-out  task  based  on  visual  cues  (circles  that 
change  color).  We  found  that  the  optimal  features  needed  to  determine  movement 
intent  occur  within  300  ms  within  the  planned  movement.  We  also  began  collecting 
data  exploring  intention  and  eye  movements  in  a  free-gazing  paradigm.  In  2010, 
further  analyses  of  the  center-out  task  and  a  similar  study  showed  that  we  could 
distinguish  left  vs.  right  movement  attention  with  very  high  accuracy,  with  new 
details  about  cortical  activity  in  Ml,  premotor  and  parietal  reach  areas.  We  launched 
a  major  new  study  in  2011,  with  eight  target  locations  and  five  experimental  stages. 
ECoG  data  showed  that  we  could  predict  movement  intention  prior  to  movement 
onset  very  well  based  largely  on  high  gamma  and  LMP  activity,  and  provided  new 
insight  about  critical  cortical  regions.  In  2012,  in  addition  to  further  analysis  of  the 
data  from  2011,  we  also  we  conducted  two  further  studies  regarding  directional 
intention.  These  major  efforts  were  extended  through  2013.  Based  on  linear 
regression  and  other  means,  we  showed  effective  discrimination  of  movement  vs. 
rest,  as  well  as  movement  intent  vs.  rest.  Further  analyses  also  found  additional  new 
information  about  cortical  regions  involved  in  movement  intention  and  the 
temporal  dynamics  of  movement  preparation  and  execution. 


Imagined  speech 

In  2009,  we  reported  that  we  collected  EEG  data  from  16  subjects  performing 
imagined  speech  tasks  across  five  different  studies.  We  extended  these  efforts  in 
2010  and  found  that  ECoG  was  much  more  effective  at  decoding  imagined  speech 
than  EEG.  Hence,  in  2011,  we  returned  to  ECoG-based  research,  collecting  data  from 
six  subjects  during  covert  and  over  speech  tasks.  We  assessed  spatial  size  and 
different  high  frequency  bands.  Interestingly,  we  found  evidence  suggesting 
different  processes  for  overt  vs.  covert  speech,  and  further  found  that  early  speech 
processing  seems  to  differ  if  people  intend  either  overt  or  covert  speech.  We 
extended  this  work  in  2012  in  the  context  of  the  well-established  Indefrey  model. 
Results  with  low-frequency  and  high-frequency  data  supported  but  also  extended 
this  model,  and  we  conducted  two  more  studies  to  further  explore  the  time  course 
of  mental  activity  associated  with  imagined  speech  and  identification  of  specific 


phonemes.  In  2013,  we  reported  new  analyses  of  the  speech  envelope,  with  details 
of  relevant  cortical  areas  including  the  superior  temporal  gyrus  and  inferior  frontal 
gyrus.  We  highlighted  original  electrophysiological  evidence  showing  the  speech 
envelope  of  non-auditory  regions  and  their  involvement  in  speech. 


Specific  Aim  Two 

Some  of  the  research  presented  in  the  preceding  section  used  ECoG  methods.  We 
recognize  that  these  methods  are  less  directly  applicable  to  field  applications. 
However,  our  work  confirmed  our  expectations  that  ECoG-based  research  could 
provide  helpful  information  that  would  improve  non-invasive  research.  For 
example,  we  noted  some  improvements  to  our  understanding  of  key  cortical  areas 
and  processes,  across  different  time  scales,  of  planning  and  executing  attentional, 
intentional  and  linguistic  tasks.  We  compared  results  from  EEG  and  ECoG  recording 
with  the  same  paradigm,  such  as  in  our  2009  comparison  that  found  similar 
activation  time  courses  in  a  language  task. 

In  this  Specific  Aim,  we  stated  that  we  would  determine  "whether  these  features  can 
be  detected  noninvasively".  Our  ECoG  work  also  helped  identify  avenues  that  are 
less  promising  for  non-invasive  work.  For  example,  we  concluded  that  EEG  activity 
is  relatively  ineffective  at  imagined  speech  and  analyses  of  complex  movement 
planning,  such  as  hand  movement  details.  Some  of  our  efforts  in  this  Specific  Aim 
were  addressed  through  modeling  efforts  across  different  years.  The  efforts  further 
supported  our  theoretical  interests,  with  considerable  synergy  between  the  time 
course  and  general  region  of  activity  for  both  ECoG  and  EEG  data,  although  EEG  data 
were  of  course  less  detailed. 


Specific  Aim  Three 

We  have  indeed  validated  the  use  of  brain  signals  for  communication  and 
orientation.  We  showed  that  ECoG  (and  sometimes  also  EEG)  can  indeed  be  used  to 
learn  about  the  user’s  auditory  and  visual  attention,  directional  orientation, 
movement  planning,  imagined  phonemes  and  other  details.  As  noted  in  Specific  Aim 
Two,  we  provided  much  more  detailed  information  about  which  brain  signals  might 
be  practical  even  with  non-invasive  means,  and  why.  The  aforementioned  modeling 
efforts  further  bolstered  our  plans  to  develop  intelligent  real-time  tools  that  can  go 
well  beyond  conventional  analyses  by  providing  the  system  with  a  more  thorough 
"understanding”  of  relevant  brain  states  and  processes. 

Our  efforts  to  validate  brain  signals  for  practical  application  were  intertwined  with 
our  work  with  the  BCI2000  software  package.  BCI2000  has  been  upgraded  and 
improved  in  numerous  ways  over  the  past  several  years.  Among  other  progress,  we 
have  added  a  newer  and  more  flexible  architecture,  support  for  many  more 
programming  environments  and  third  party  software,  integration  of  third  party 
tools,  improved  documentation,  additional  software  modules  for  new  options,  and 
improved  capabilities  for  real-time  processing  under  more  challenging  conditions 


(such  as  simultaneously  recording  two  types  of  brain  signals,  or  other  inputs). 
BCI2000  is  much  better  suited  than  other  platforms  when  recording  from  a  suite  of 
biophysical  signals,  including  activity  from  the  brain,  eyes,  muscles,  heart, 
respiration  and  other  signals.  BCI2000  has  become  the  de-facto  standard  for  real¬ 
time  biosignal  acquisition  and  analysis  in  BCI  research,  and  is  gaining  attention  in 
related  fields. 

In  particular,  BCI2000  has  grown  into  a  platform  capable  of  clinical  (as  well  as 
academic  and  commercial)  research.  BCI2000  has  been  very  extensively  validated  in 
hospital  environments  with  numerous  patients,  through  the  work  in  this  MURI 
project  and  elsewhere.  We  remain  very  active  in  collaborations  with  clinicians  in 
several  countries  who  are  using  BCI2000  with  patients,  and  feedback  from  end 
users  continues  to  indicate  that  BCI2000  provides  major  advantages. 

Collaboration  and  dissemination 

The  efforts  in  this  MURI  project  involved  substantial  collaboration  with  top  clinical 
and  research  experts.  External  collaborators  during  the  project  include  Dr.  Eric 
Leuthardt  from  the  University  of  Washington,  Dr.  Christoph  Guger  from  g.tec,  Dr. 
Bijan  Pesaran  from  New  York  University,  and  Dr.  Kai  Miller  from  Stanford 
University.  We  have  ongoing  collaborations  with  Dr.  Cuntai  Guan  from  the  Institute 
for  Infocomm  Research  in  Singapore,  Dr.  Dean  Krusienski  from  Old  Dominion 
University,  and  Dr.  Fabien  Lotte  from  INRIA  Bordeaux. 

Results  from  this  project  have  been  published  in  high-impact  peer-reviewed 
journals  including  Neuroimage,  the  Journal  of  Neuroscience,  Frontiers  in 
Neuroprosthetics,  Frontiers  in  Human  Neuroscience,  PLoS  One,  Experimental  Brain 
Research,  and  the  Journal  of  Neural  Engineering.  In  2011,  project  work  was  selected 
as  a  finalist  for  the  annual  BCI  Award,  leading  to  a  book  chapter  describing  our  work 
with  Springer  Publishing.  In  2014,  two  different  project-related  studies  were 
nominated  as  finalists  (i.e.,  top  ten  of  close  to  100  submissions)  for  the  annual  BCI 
Award.  We  have  also  produced  other  book  chapters  with  project  results,  along  with 
dozens  of  conference  presentations.  We  also  disseminated  the  MURI  project  through 
several  BCI2000  and  ECoG  workshops,  which  attracted  students,  doctors,  engineers 
and  other  attendees. 

Project  results  have  also  been  disseminated  in  different  ways  by  third  parties  who 
were  not  involved  in  this  project.  Our  BCI2000  software  has  been  used  by  hundreds 
of  groups  around  the  world,  has  been  evaluated  by  the  Army  Research  Laboratory  in 
Aberdeen,  and  by  investigators  of  the  partner  MURI  (PI  Dr.  D'Zmura,  Dr.  Srinivasan). 
Several  Workshop  Tours  conducted  by  g.tec  (the  top  manufacturer  of  BCI  hardware) 
have  presented  project  achievements  as  promising  uses  of  g.tec  hardware.  Project 
accomplishments  were  highlighted  in  the  Future  BNCI  roadmap  (commissioned  by 
the  EC)  as  examples  of  promising  BCI  research  from  the  USA,  and  have  been 
references  in  several  peer-reviewed  publications. 

In  summary,  this  section  provides  another  means  of  verifying  the  success  of  our 
project  -  respect  from  peers.  Our  colleagues’  respect  for  our  achievements  is 
reflected  through  numerous  different  means: 


1)  Prior  and  ongoing  collaborations  with  top  research  groups; 

2)  Several  peer-reviewed  publications  in  top  journals; 

3)  Dozens  of  citations  of  these  publications; 

4)  Adoption  of  BCI2000  and  other  methods  by  different  groups; 

5)  Participation  in  BCI2000  and  ECoG  workshops  highlighting  project  results; 
and 

6)  Other  dissemination  of  project  accomplishments  by  third  parties. 


Conclusions 

We  have  been  very  successful  in  attaining  our  aims,  providing  new  insights  into 
attention,  directional  orientation  and  imagined  speech.  While  some  challenges  in 
developing  practical,  mainstream  brain-based  tools  are  still  significant,  we  have 
made  major  progress  toward  this  futuristic  aim.  We  have  soundly  addressed 
previously  daunting  obstacles  such  as  a  truly  practical,  flexible  real-time  software 
system  that  can  work  with  multimodal  data  and  different  devices.  Our  summary 
here  does  not  at  all  detail  our  entire  accomplishments  in  this  project. 


Our  results  will  facilitate  the  direct  assessment  of  a  soldier’s  directional  orientation, 
such  as  the  locus  of  visual  or  auditory  spatial  attention  or  the  direction  of  intended 
movements.  These  achievements  will  lead  to  new  combat  capabilities  and 
applications.  Our  results  with  imagined  speech  should  create  new  ways  to 
communicate  without  any  overt  movement.  The  resulting  novel  communications 
opportunities,  aside  from  tactical  combat  implications,  could  also  benefit  battlefield 
treatment  and  covert  operations.  This  MURI  project  could  also  impact  many  related 
domains,  including  aviation,  security,  medicine  and  psychology.  Moreover,  we  have 
developed  new  methods  to  analyze  the  brain,  and  provided  new  insights  in  neural 
function,  that  should  continue  to  provide  strong  opportunities  for  future  research. 


